feat(nzbdav): two-phase migration — import + symlink rewrite#522
Merged
feat(nzbdav): two-phase migration — import + symlink rewrite#522
Conversation
Adds a generic import_migrations table to track two-phase migration state (Phase 1 = import NZBs into AltMount; Phase 2 = rewrite arr library symlinks). Includes goose migrations for SQLite and PostgreSQL, ImportMigration model + status consts, a full ImportMigrationRepository with Upsert/MarkImported/MarkFailed/MarkSymlinksMigrated/LookupByExternalID/ ListByStatus/Stats/ExistsForSource/BackfillFromImportQueue, and wires MigrationRepo into the DB struct.
…avIds Delete HandleIDMetadataLinks and id_linker.go; remove UpdateIDSymlink and RemoveIDSymlink methods from MetadataService; remove the .id sidecar write block from WriteFileMetadata (read path preserved for Phase 2 compatibility); remove FilterExistingNzbdavIds from QueueRepository, Repository, and the BatchQueueAdder interface; remove related call sites in nzbfilesystem MOVE handler and nzbdav scanner processBatch.
- Add MigrationRecorder interface to scanner package with UpsertMigration and IsMigrationCompleted methods - Update NzbDavImporter to accept MigrationRecorder as second constructor parameter alongside BatchQueueAdder - processBatch now checks IsMigrationCompleted before enqueueing (skip already-imported/symlinks_migrated items) and calls UpsertMigration for new items, then strips nzbdav_id from the queue item metadata leaving only extracted_files if present - createNzbFileAndPrepareItem sets SkipArrNotification=true on all items - batchQueueAdapterForImporter gains migrationRepo field and implements MigrationRecorder via ImportMigrationRepository.Upsert/LookupByExternalID
…lure Wire s.database.MigrationRepo into handleProcessingSuccess and handleProcessingFailure so that import_migrations rows are marked imported/failed when the corresponding queue item finishes. Both calls are non-fatal: failures are logged as warnings and do not affect the main processing outcome. If no matching migration row exists (non-nzbdav import), the UPDATE simply affects 0 rows.
Adds RewriteLibrarySymlinks in internal/importer/migration to walk a library directory and atomically rewrite arr symlinks that point at <nzbdav_mount>/.ids/<guid> to the final altmount path. Introduces DBSymlinkLookup adapter in internal/database, a new POST /import/nzbdav/migrate-symlinks handler, and enriches the existing status endpoint with migration_stats when available.
Phase 1: import_migrations DB table tracks every nzbdav import by GUID, surviving queue resets. Scanner uses it for dedup (replaces FilterExistingNzbdavIds), sets SkipArrNotification=true, and strips nzbdav_id from queue metadata. Service marks rows imported/failed on queue item completion. Phase 2: POST /import/nzbdav/migrate-symlinks walks an arr library, detects symlinks targeting <nzbdav_mount>/.ids/<guid>, looks up the AltMount final_path in import_migrations, and atomically rewrites them. BackfillFromImportQueue seeds the table from historical queue rows so already-migrated users can skip directly to Phase 2. Cleanup: id_linker.go deleted, HandleIDMetadataLinks call removed, all .ids/ write paths removed from metadata service (read path preserved for backward compat), FilterExistingNzbdavIds dropped. Frontend: Phase 2 card with dry-run preview and Apply button appears after Phase 1 completes or when migration_stats.imported > 0.
Users who already migrated via the old nzbdav flow can now access the symlink rewrite tool directly from a top-level 'Migrate Symlinks' tab without needing to run Phase 1 first.
… add Clear All Migrations - Phase 2 symlink rewrite inside the NZBDav import flow is now gated on an empty queue (total_queued + total_processing == 0) with a live "Waiting for queue to finish" banner so users don't rewrite symlinks while imports are still in flight. - Split the NZBDav section into nested tabs (Import / Migrate Symlinks). The standalone tab runs against already-imported rows for users who imported previously and only need to rewrite the arr library. - Extract the symlink form into a shared SymlinkMigrationForm component reused by both entry points. - Add DELETE /api/import/nzbdav/migrations + UI button to wipe all nzbdav migration rows, for the case where the user deleted imported files from AltMount and wants to force a full re-import.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
import_migrationstable tracks every nzbdav import by GUID, surviving queue resets. Scanner uses it for dedup (replacesFilterExistingNzbdavIds), setsSkipArrNotification=true, and stripsnzbdav_idfrom queue metadata. Service marks rowsimported/failedon queue completion.POST /import/nzbdav/migrate-symlinkswalks an arr library, detects symlinks targeting<nzbdav_mount>/.ids/<guid>, looks up the AltMountfinal_pathinimport_migrations, and atomically rewrites them.BackfillFromImportQueueseeds the table from historical queue rows so already-migrated users can jump directly to Phase 2.id_linker.godeleted,HandleIDMetadataLinkscall removed, all.ids/write paths removed frommetadata/service.go(read path preserved for backward compat),FilterExistingNzbdavIdsdropped from queue repo.migration_stats.imported > 0.Test plan
go test -race ./internal/...passesbun run check && bun run buildpassesimport_migrationsrows flippending → imported, no.ids/symlinks created, arr not notifiedreadlinklibrary files confirm new targets point to AltMount mount pathimport_queue.metadata.nzbdav_idrows but noimport_migrations, hit Phase 2 endpoint → backfill kicks in and symlinks get rewrittenGET /import/nzbdav/statusincludesmigration_statsfield